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GAME CONSOLE BASED DIGITAL PHOTO ALBUM 

This application is a continuation in part of Serial No. 09/465,982, filed 
on December 16, 1999. 

5 FIELD OF THE INVENTION 

The present invention relates generally to field of image retrieval and 
organization. More specifically, the present invention is directed to multimedia 
creation using digital objects (e.g., images). 

10 BACKGROUND 

0 Photographs play a central role in many types of informal storytelling. 

One of the most common and enjoyable uses for photographs is to share stories 
3 about experiences, travels, friends and family. Almost everyone has experience 

v - 1 
V *[ 

*P with this form of storytelling, which ranges from the exchange of personal 

y 15 reminiscences to family and cultural histories. The World Wide Web can 



fSl facilitate the sharing of such stories in digital form and has inspired a 

O movement towards "digital storytelling/' Stories in digital form are referred to 

herein as digital stories. Digital photographs have an advantage over print 
photographs in that users can search for and retrieve them both by their content 
20 (e.g., features such as color and texture) and by their metadata (e.g., user- 
supplied text annotations). Today, most digital stories are created by people 
with computer skills using special-purpose software for editing images and 
authoring Web pages. Furthermore, these digital stories are created on systems 
relying on graphical interfaces dependent on windows, or tables, and some 
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type of pointing or cursor control device. Such systems are typically not 
portable and are complex to use. These features make these systems difficult to 
share. These systems also require significant training and are inadequate for 
mass consumer use. 
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SUMMARY OF THE INVENTION 

A method and a system that combine capabilities for storing, authoring, 
and viewing various forms of digital media are described. In one embodiment, 
the system comprises means for accepting a cartridge having a slot for a 
5 memory device to store a plurality of media objects, means for displaying the 
plurality of media objects in reduced visual representations in at least one track, 
and means for navigating among the reduced visual representations. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
The present invention is illustrated by way of example in the following 
drawings in which like references indicate similar elements. The following 
drawings disclose various embodiments of the present invention for purposes 
of illustration only and are not intended to limit the scope of the invention. 

Figure 1 illustrates an exemplary embodiment of the display screen of 
one embodiment of a digital story-telling system. 

Figure 2 illustrates an exemplary embodiment of control buttons of the 
digital story-telling system. 

Figure 3 illustrates an exemplary expand / collapse view of a story. 

Figure 4 illustrates an exemplary XML metadata file that the system 
reads upon initialization of the system. 

Figure 5A illustrates an exemplary XML metadata for a story. 

Figure 5B illustrates another exemplary XML metadata file for a story. 

Figure 6 illustrates an exemplary XML metadata file for a single 
photograph. 

Figure 7 illustrates an exemplary computer system. 
Figure 8 illustrates an exemplary console having similar capabilities as 
the digital story-telling system. 
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DETAILED DESCRIPTION 



A multimedia story creation and playback system using a game console 
5 is disclosed. The following detailed description sets forth numerous specific 
details to provide a thorough understanding of the invention. However, those 
of ordinary skill in the art will appreciate that the invention may be practiced 
without these specific details. In other instances, well-known methods, 
procedures, protocols, components, algorithms, and circuits have not been 
55 10 described in detail so as not to obscure the invention. 

TKS? 

DO Some portions of the detailed descriptions that follow are presented in 

P terms of algorithms and symbolic representations of operations on data bits 

^ within a computer memory. These algorithmic descriptions and 

m representations are the means used by those skilled in the data processing arts 

ill 15 to most effectively convey the substance of their work to others skilled in the 
•= art. An algorithm is here, and generally, conceived to be a self-consistent 

sequence of steps leading to a desired result. The steps are those requiring 
physical manipulations of physical quantities. Usually, though not necessarily, 
these quantities take the form of electrical or magnetic signals capable of being 
20 stored, transferred, combined, compared, and otherwise manipulated. It has 
proven convenient at times, principally for reasons of common usage, to refer 
to these signals as bits, values, elements, symbols, characters, terms, numbers, 
or the like. 
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It should be borne in mind, however, that all of these and similar terms 
are to be associated with the appropriate physical quantities and are merely 
convenient labels applied to these quantities. Unless specifically stated 
otherwise as apparent from the following discussion, it is appreciated that 
5 throughout the description, discussions utilizing terms such as "processing" or 
"computing" or "calculating" or "determining" or "displaying" or the like, refer 
to the action and processes of a computer system, or similar electronic 
computing device, that manipulates and transforms data represented as 
physical (electronic) quantities within the computer system's registers and 

3 1: 

* 10 memories into other data similarly represented as physical quantities within the 
% computer system memories or registers or other such information storage, 

f|j transmission or display devices. 

s The present invention also relates to apparatus for performing the 

* " 

2 operations herein. This apparatus may be specially constructed for the required 

pj 15 purposes, or it may comprise a general purpose computer selectively activated 
or reconfigured by a computer program stored in the computer. Such a 
computer program may be stored in a computer readable storage medium, 
such as, but is not limited to, any type of disk including floppy disks, optical 
disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), 
20 random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical 

cards, or any type of media suitable for storing electronic instructions, and each 
coupled to a computer system bus. 



Ricoh 



07445 1.P119X 



The algorithms and displays presented herein are not inherently related 
to any particular computer or other apparatus. Various general-purpose 
systems may be used with programs in accordance with the teachings herein, or 
it may prove convenient to construct more specialized apparatus to perform the 
required method steps. The required structure for a variety of these systems 
will appear from the description below. In addition, the present invention is 
not described with reference to any particular programming language. It will 
be appreciated that a variety of programming languages may be used to 
implement the teachings of the invention as described herein. 

A multimedia storytelling system that provides a user capabilities to 
share digital photographs and stories. In one embodiment, the system allows 
the user to seamlessly switch between browsing, viewing, authoring, and 
playing back the photographs and/ or stories. In one embodiment, a 
multimedia storytelling system includes three components: a storage 
component, a display component, and an interaction component. The storage 
component stores digital media objects which are displayed by the display 
component. The system provides the interaction component (e.g., control 
buttons) that allow a user to navigate the digital media objects and create 
stories, or playlists. For purposes herein, a story, or a "playlist", is an ordered 
collection of digital media objects, such as, for example, images or video clips, 
with one or more narration tracks. Imported stories on the top track may be 
merely a single photograph with or without a narration track. Photographs 
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may be imported in batches, similar to that of a roll of film. Similarly, a 
collection may comprise the images of multiple pages in a single document. 
Thus, although there may not be an associated narration track, an imported 
collection of photographs, or other media objects, or a single photograph may 
represent a story for purposes herein. 

In one embodiment, the digital media stored in the system includes 
photographs, images, audio and video clips. The digital media may include 
raw objects, such as, for example individual photographs, as well as authored 
objects that combine multiple forms of media into integrated stories. An 
example of an authored object is a sequence of photographs with an audio, or 
narration, clip associated with or accompanying the photographs. In one 
embodiment, the time of capture for an object is known and is stored along 
with the object in the system. 

New objects may be input into the system automatically without 
requiring any action from the user. New objects are input into the system using 
one or more sources, such as, for example, on flash memory cards containing 
digital photographs, video capture camera, reading digital photographs or 
video clips from floppy or CD-ROM drives, network (e.g., Web) downloads, 
etc. In one embodiment, the user inserts a flash memory card into a slot in the 
system and the photographs from the flash memory card are automatically 
copied and stored in the system. In another embodiment, the system is able to 
receive email messages with attached photographs and stories. The 
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photographs and/ or stories are then automatically accessible. It would be 
apparent to one skilled in the art the various ways multimedia objects may be 

input into the system. 

A narration may take a variety of forms, such as, for example, recorded 
audio, digital ink, or typed notes. In one embodiment, a narration track may 
apply to a single object or to a number of objects (e.g., a narration track for a 
sequence of images). There may be multiple narration tracks for a single object. 
That is, the same digital object (photograph) may have several associated 
narrations. 

The display component of the system provides multiple tracks 
displaying digital media objects to the user. A first track displays digital media 
objects that are stored in the system. A second track displays digital media 
objects that have been integrated into authored stories. In one embodiment, a 
third track displays one or more digital media objects associated with a story 
currently being authored using the system. 

The display component of the system also allows the user to navigate 
through the objects in storage. This may include being able to browse through 
raw objects (e.g., objects without narrations associated or attached thereto), 
browse through stories, and play back previously created stories. In one 
embodiment, playing back a story involves showing the user a slideshow of 
images along with any accompanying narrations. 
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The interaction component of the system allows the user to create new 
stories. In one embodiment, the user may create new stories using one or more 
of several approaches. For example, a user may author a story by starting with 
an empty or blank story, selecting objects to add to the story, and adding 
5 narrations to individual objects or a range of objects. The selection of objects 
and the addition of narrations may be performed in any order. A user may 
author a new story by performing edit or delete operations on an existing story. 

In one embodiment, the interface is a joystick or other game console 
controls. In another embodiment, the system has a modeless interface, giving 
10 the user an interface of a small number of buttons and no pointing device or 
touchscreen. In another embodiment, the system may be implemented on a 
typical desktop computer or on a standalone appliance. The user controls for 
such a system may utilize standard input devices, such as, for example, a 
mouse, a keyboard, a touchscreen, a touch pad, or a physical control 
1 5 environment of buttons and / or sliders designed specifically for the interaction 
described herein. 

Figure 1 illustrates one embodiment of a display screen and interface of a 
system. Media objects may be loaded onto the system from various sources, 
such as, for example, digital cameras, digital video capture, microphones, 
20 scanners, or may be created on the system itself. These objects are then stored 
in the system on a storage medium (e.g., random access memory, a hard drive, 
flash memory). 
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As illustrated in Figure 1, the system provides a visual interface 100 
which splits the screen into three general areas. In one embodiment, the first 
area 102 at the top of the screen, the second, or central, area 103 at the bottom 
right side of the screen, and the third area 104 at the bottom left side of the 
screen. 

The first area 102 provides a graphical representation for browsing and 
navigating through media objects (e.g., photographs, etc.). In one embodiment, 
there are three horizontal tracks of thumbnail images, tracks 105, 110 and 115, 
each of which can be navigated by scrolling. 

In one embodiment, the top track 105 shows images of existing 
photographs ordered by time of creation (or other such features). The time of 
creation may be indicated by time stamps. These may be grouped into "rolls of 
film" or other natural clusters which are called imported stories. In the case of 
scanned prints, imported stories may correspond to literal rolls of film. In the 
case of digital photographs, the photographs may correspond to a set of 
photographs downloaded from the camera in one session. The top track 105 
may display all or many existing photographs currently stored in the system. 
That is, any image stored on the system may be displayed. 

In one embodiment, the imported stories on the top track 105 may be 
generated automatically as a result of a database query. For example, a 
database query for "all images stored on a device containing blue" returns 
images having the "blue" identifier. The resulting images are then imported 
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into the system. Note that there is no requirement that the first track 105 
include any stories at all. That is, the first track 105 may contain zero or more 
stories, each of which having one or more images. 

Within an imported story, the photographs may be ordered 
chronologically. Photographs from digital cameras may be ordered according 
to when they were taken, while images scanned from print photographs are 
ordered by scanning time. In one embodiment, in order to distinguish separate 
stories, the system uses alternating background colors with each color 
corresponding to different story. 

The middle, or second, track 110 contains authored stories, each 
including a sequence of one or more photographs selected by an individual. 
The middle track may also contain authored stories sent to this system 
electronically from some other device. Authored stories can be distinguished 
from raw imported stories by having a creator, sender, narration or some other 
features. Each story appears as a sequence of thumbnail images. In one 
embodiment, the imported and authored stories are ordered according to their 
time of creation with separate stories being visually distinguished using 
different colored backgrounds. In another embodiment, stories in a track may 
be separated by spaces in the display. The size of the spaces may depend on 
the length of time between the story creation times. For example, in one 
embodiment, photographs taken on the same day are closer together on the 
second track 110 than those separated by a day or a weekend. The space may 
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be a function of a time duration, but not necessarily linear. For example, there 
might be set increments for "day", "weekend", "week" and "more than a 
week". 

The bottom, or third, track 115 represents a story being authored. That 
is, the bottom track 115 includes a working set of thumbnail images that have 
been selected and manipulated by the user. The set of images may only appear 
in the bottom track 115 for the current session with the system. If the user 
wishes to further author a story after a session has ended, the user selects the 
authored story for inclusion into the bottom track 115. 

In one embodiment, a thumbnail image appears in the bottom track 115 
if it has been added to the working set by pressing one or more control buttons 
(e.g., either of the + (add) or record buttons, as detailed below). At any time, 
the story being authored has a pending status. When the story is completed 
and saved by the user, it joins the set of authored stories in the middle track 
110. In one embodiment, the middle track 110 and the bottom track 115 need 
not be visually distinguished, nor for that matter, any of the tracks. 

The display of the three tracks 105, 110, and 115 enables an essentially 
modeless interface where a user can simultaneously view authored stories, 
view/ navigate through photographs and view/ create new stories. The display 
also provides helpful context for viewing the current image. For example, 
when the user selects a thumbnail and views an image from one story, adds it 
to the current story, then adds annotation, the context of that image with the 
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surrounding thumbnails in both the original and current story remains visible 
and helpful for annotation. 

In one embodiment, each thumbnail image appears exactly once in the 
top track 105. It may appear in any number of stories in the middle track 110 
5 and appears in the bottom track 115 only if it has been explicitly added to or 
annotated as part of the story currently being authored. 

In a typical interaction, the user comes across a thumbnail image and 
adds it to the working set (bottom track 115). The users may also record a 
related voice narration to accompany the selected thumbnail image. At the end 
10 of the session, all the thumbnail images in the bottom track 115 are grouped 

into a single story, and the story is then appended or added to the middle track 



110. 



The central area 103 of the screen display shows a large image 120. The 



large image 120 corresponds to a thumbnail image of a story or photograph 
15 currently being selected in the tracks in the first area 102. This allows the user 
to view photographs at a size that is comfortable and enjoyable (as opposed to 
the smaller thumbnail versions in tracks 105, 110 and 115). In another 
embodiment, other objects may require other display characteristics. 

The third area 104 of the screen display shows the available audio 
20 narrations of the photograph currently displayed in the central area 103. In 
another embodiment, other information, such as, for example, creation date 
may also be displayed in the third area 104. As shown in Figure 1, there are 
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two narrations 130 and 135 available for the photograph 120 in the central area 
103 indicated by the depiction of the wave signal. In one embodiment, each 
narration has a visual display corresponding to the length of time of the 
narration (e.g., the longer the signal wave the longer the narration), together 
5 with related recording information, such as, for example, time and date 
recorded, and the name of recording user. 

Figure 2 illustrates an exemplary embodiment of the control buttons and 
the display of a system for creating and/or viewing multimedia stories. In one 
embodiment, the control buttons are integrated into and positioned at the sides 
10 of the body of the system to enable the user manipulate the controls while 
holding the system in two hands. This further removes the need for a 
keyboard, mouse, or other device that forces the user to let go of the system. 

As illustrated in Figure 2, the control buttons in this embodiment include 
navigation buttons such as a scroll left button 205 to scroll the images in a 
15 selected track to the left, a scroll right button 210 to scroll the images in a 

selected track to the right, and a track selection button 215 to select the track to 
scroll. In one embodiment, a jog shuttle is used to control scrolling. 

In one embodiment, a colored (e.g., yellow) vertical line 255 indicates the 
selected track and the selected thumbnail image of this track. The vertical line 
20 255 is shown in the first thumbnail image of the bottom track. The selected 
thumbnail image corresponds to the large image displayed in the central area 
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250. The track selection button 215 moves the vertical line 255 between the 

three different tracks. 

When either of the scroll buttons 205 or 210 is pressed, the new selected 
thumbnail image on the current track either to the left or to the right side of the 
previously selected thumbnail image is selected and displayed in the central 
area 250. As a different thumbnail image moves under the vertical line 255, the 
corresponding image is displayed in the central area 250. Note that any type of 
indicator or indication may be used on the screen to select an image. It should 
also be noted that while in one embodiment, the scroll button 205 or 210 causes 
the images in the track with the vertical line 255 to move to the left or right 
while leaving the vertical line 255 in place, the system could keep the images in 
place while moving the vertical line 255 to select an image. In one embodiment, 
the navigation between the images may be controlled by an external joystick 
communicating with the system through a joystick interface. 

In one embodiment, the user may quickly traverse the thumbnail images 
on a track by using variable-speed scrolling. At standard speed, the display 
appears as shown in Figure 2. In one embodiment, standard speed refers to 
pressing one time on the left scroll button 205 or the right scroll button 210. 
This moves the track by exactly one thumbnail image. Faster scrolling speeds 
may be enabled by rendering low resolution versions of the thumbnail images 
that are quickly accessible from a separate index in the system and by not 
rendering the image in the central area 250 or the narrations in the audio area 
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260. In one embodiment, such images may be cached in memory to increase 
access speed. To increase scrolling speed, the user may need to hold down a 
scroll buttons 205 or 210 for a longer period of time. When this occurs, the 
images appear to be scrolling faster by the vertical line indicator 255. 
5 In one embodiment, an expand/ collapse button 265 controls the view or 

the number of photographs displayed for each story on one of the three tracks. 
In the expanded view, every thumbnail image in a story is shown. In one 
embodiment, the expanded view is the default view. In the collapsed view, 
each story is represented by a predetermined number of thumbnail images 
fi 10 (e.g., one thumbnail image). In one embodiment, a first thumbnail image of the 
2 story is used to represent the story. The collapsed view allows for faster 

W navigation. Figure 3 illustrates an exemplary expand/ collapse view of a story. 
%, Referring to Figure 3, the two stories 305 and 310, with thumbnail images 

pi"*!" 

n belonging to one story having the same colored background, are shown in 

?. In! 

*r li 

O 15 expanded form. The same two stories are illustrated in collapsed form in 

thumbnails 315 and 320, with the first thumbnail image of each story 305 and 
310 being used as the thumbnail image in the corresponding collapsed form. 

Referring back to Figure 2, the cluster of buttons at the bottom left of the 
system 200 provides the user with additional controls for authoring, browsing 
20 and playing back of stories. A play button 220 is used to enter a play mode to 
allow an individual to view existing stories. During the play mode, the system 
starts playback from the currently selected thumbnail image. In one 
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embodiment, although not necessary, each thumbnail image has an audio clip 
associated with it as part of a story in which it resides. Furthermore, each 
thumbnail image may be part of more than one story, and thus may have more 
than one audio clip associated with it, one audio clip for each story. However, 
5 an audio clip may or may not contain any recorded narration. When a 

narration exists, the narration is played through a sound system which may be 
part of the game console or part of the display device. If there is no recorded 
narration, there will be a pause of a certain length of time, for example one 
second. After the narration is played or after the pause (due to not having the 

r, :: 

fi 10 narration), the selected track automatically scrolls forward to a next thumbnail 

fixZ ii 

imp ^ 

3 image in the story. In one embodiment, the next thumbnail image is 

W determined based on a default viewing and storytelling direction of left-to- 

)U right. If the user navigates to a new thumbnail image while the system is 

ffli 

£J playing, for example scrolling left or right, changing to a different track, 

5 15 playback of the currently playing audio clip is stopped. If the user takes no 

further action for a brief length of time (e.g., one second), the system plays the 
audio clip associated with the newly selected thumbnail image and continues 
with that story from that point on. 

As discussed above, the selected thumbnail image may have more than 
20 one associated audio clip and corresponding narrations. Whenever a thumbnail 
image is selected, all of the audio narrations associated with that thumbnail 
image are displayed in the audio area 260. In one embodiment, if the selected 
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thumbnail image has more than one associated audio clip, and when the system 
is playing an audio clip, pressing the play button 220 causes the system to 

advance to a next audio clip. 

Referring to Figure 1, there are two available narrations for the selected 
5 thumbnail image, narration 120 and narration 125. Each narration is marked 
with the time of recording and the name of the recording user. The length of 
the wavy lines is proportional to the duration of the audio. The narration 
associated with the selected story is listed first, as in narration 120. Referring to 
Figure 2, the narration associated with the selected story is played by default 
10 when the play button 220 is pressed. Pressing the play button 220 multiple 
times in quick succession selects one of the alternate audio clips and playback 
"jumps" to the corresponding story, providing a method of automatic 
hyperlinking between stories. 

The system remains in the play mode until the stop button 225 is pressed 
1 5 or until the end of story is reached. When this occurs, the system enters into 
stop mode. While in the stop mode, no audio clip is played by the system. 

In one embodiment, while the system is in the play mode, the remove (-) 
button 235 has no effect on the play mode of the system. However, pressing the 
add (+) button 230 adds the currently displayed image to the working set. 
20 Thus, the system appends the currently displayed image to the working set, but 
it does not stop the play mode. Furthermore, if the user presses the save button 
245 while the system is in the play mode, the system stops playing the audio 
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clip and performs a save operation. Furthermore, if the user presses the record 
button 240 while the system is in the play mode, the audio clip is stopped. 

In one embodiment, while in the authoring mode (not the play mode) a 
story may be created by pressing the + (add) button 230 when the vertical line 

5 255 is positioned over a particular thumbnail image to append a copy of that 
currently selected thumbnail image onto the working set or the bottom track 
115. The - (remove) button 235, conversely, removes the selected thumbnail 
image (i.e., the image over which the vertical line 255 is positioned) from the 
working set on the bottom track 115. 

10 While in authoring mode, pressing the record button 240 starts the 

recording function. While the recording function is active, the audio recorded 
by the microphone is stored and associated with the currently displayed 
thumbnail image in the working track. In one embodiment, this is done by 
making a link in the underlying XML file. If the thumbnail image is not already 

15 in the working set, it is appended before the recording begins, as though the + 
button 230 was pushed first. If the selected thumbnail image is already on the 
working track, the new recording overwrites any previous recording associated 
with the thumbnail image in the story being authored. While recording, if the 
user selects a new thumbnail image, such as, for example by scrolling left or 

20 right, by changing to a different track, or by pressing the expand/ collapse 
button, the recording continues for the audio clip associated with the new 
thumbnail image. In addition, the new thumbnail image is appended to the 
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working track, if it is viewed for more than a short time, (e.g. 1 second). 
However, if the user scrolls left or backward, but still remains on the working 
track, recording is automatically stopped. This prevents accidental erasures. In 
other embodiments, the new thumbnail image is inserted at a point before or 
after the thumbnail image last selected on the working track (e.g., the center 
image of track 3). In this situation, recording continues starting with the audio 
clip associated with the new inserted thumbnail image. Furthermore, pressing 
the play button 220 also stops the recording mode and put the system in the 
play mode. This makes recording a story as similar to viewing a story as 
possible. 

In another embodiment, the system may include a touchscreen or a 
pointing device (e.g., cursor control device). In this case, while in the recording 
mode, the user's pointing gestures would be captured. During playback, areas 
that had been pointed to are highlighted at appropriate times. One way to 
perform highlighting is to darken the rest of the image. Alternatively, a semi- 
transparent overlay can be used (e.g., a yellow circle). 

When the system is in the stop mode, the user may drag thumbnail 
images from one track to another, or the user may drag thumbnail images to 
different points within a story. Alternatively, a dragging operation can be used 
in place of scroll left/right to move all or multiple images in a track at the same 
time. 
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Pointing to a particular thumbnail selects the thumbnail image, selects 
the track the thumbnail image is on, and displays the corresponding large 
image in the central area 250. In another embodiment, touching the screen may 
highlight an image in a track but not select it. 

In one embodiment, the recording operation supports both "select then 
narrate " and "select while narrating" strategies. In "select then narrate" 
strategy, the users may compose a story by first selecting a working set of 
thumbnail images using the + button 230 or the - button 235 and then 
annotating each thumbnail images in order. Alternatively, in the "select while 
narrating" strategy, the users may continuously record narrations while 
navigating and selecting photos. When recording is active, each new 
photograph that a user views for longer than a short time interval is 
automatically added to the working set along with any recorded input. This 
supports the "select while narrating" strategy. For sound recording, in one 
embodiment, a microphone 270 is attached to or integrated into the external 
body of the system. In another embodiment, the device is equipped with a 
speech recognition software to translate audio narration into text. 

The group of buttons at the bottom right of Figure 2 controls story 
operations. The save button 245 "saves" the current working story that is 
displayed on the bottom track by moving it to the end of the middle track. 
Note that the current state of the system is also saved to the system storage 
device at that time. The XML files (or equivalent representations) along with 
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audio clips may be stored in memory (e.g., RAM) and written out to a hard disk 
when saved. (They may also be written out at other times.) In another 
embodiment, the user would also have the option of electronically sending a 
completed story to another user for viewing on a similar system or on a regular 
5 PC via a media player application or standard Web browser. In another 
embodiment, a "print" button allows the user to print a selected image on a 
locally connected, via wire or wireless technology, printer or a remote device. 

In one embodiment, the system has an attached image 
creation/capturing device (e.g., a video camera). The video camera can point 
* 10 inward at the holder of the system or it can point outward. A take-picture 
5 button on the system allows the user to take a still image from the camera, time- 

m stamp, and add the image to the top track. In another embodiment, an image of 

*_ the narrator is automatically grabbed by the camera at the beginning of every 

? recording; session, or at pre-set intervals, or every time the video input changes 

S 15 by more than some predetermined threshold. Using the video camera, the 

images captured by the video camera may be added to the first track 105 and 
available for inclusion in authored stories like all other objects in the system. 

In one embodiment, stories and metadata about photographs are stored 
on the storage device in Extensible Markup Language (XML). Figure 4 
20 illustrates an exemplary XML metadata file that the system reads during 
initialization. The initializing metadata file 400 is stored in a predetermined 
known location on the storage device. In one embodiment, the initializing 
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metadata file is named "families.xml" and the location of the file is determined 
using a local Uniform Resource Locator (URL) corresponding to the file on the 
storage device of the system. For each possible group (e.g., family) using the 
system, the initializing metadata file contains pointers to the stories 
corresponding to the three tracks. For example, the metadata file of Figure 4 
points to a story file 405 "meta6.xml". In one embodiment, the initializing 
metadata file also maintains further administrative functions, such as, for 
example, user ids and passwords to prevent unauthorized viewing of personal 
stories. In one embodiment, the initialization file contains user specific 
information. In another embodiment, the system assumes a single user. 

Figure 5A illustrates an exemplary XML metadata for a story. The XML 
metadata file for story 500 points to two other stories, "metaS.xml" 505 and 
"meta5.xml" 510. Note that each metadata file has an associated "type" 
attribute having a value of either "story" or "document" indicates single object. 
Figure 5B illustrates another exemplary XML metadata file for a story. 
Metadata file 520 represents the story "meta3.xml" 505 pointed to by the 
metadata file 500. Metadata file 520 has a list of photographs shown as 
"metal. xml" 525 and "meta2.xml" 530. Note that the photograph references are 
in the form of the URL of the corresponding metadata XML file. Each 
photograph may also have an associated audio clip shown as "clipl.au" 535 
and "clip2.au" 540. In one embodiment, the offset attribute of the audio tags 
associated with audio clip 535 or audio clip 540 specifies a starting location of 
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the referenced audio narration (such as when, for example, one audio file 
contains more than one user-supplied audio narration) within the audio file and 

is measured in milliseconds. 

Figure 6 illustrates an exemplary XML metadata file for a single 
5 photograph. Metafile 600 represents the image "metal .xml" pointed to by the 
metafile illustrated in Figure 5B. Metafile 600 includes URLs that point to two 
image files, a full size image 605 that may be used in area 103 of the display and 
a thumbnail image 610 that may be used in the tracks 105, 110, and 115. In 
addition, the metafile 600 includes color data for a reduced representation. The 
10 reduced representation of the image may be rendered either as a thumbnail 
^ image or as a full size. 

m In one embodiment, using XML allows for easy translation to other 

formats such as Hypertext Markup Language (HTML) or Synchronized 
Jj Multimedia Integration Language (SMIL). This enables stories to be shared 

H 15 with others and viewed on different devices. The SMIL format is especially 

appropriate as it allows the synchronization of audio with a series of images to 
match the structure of the stories. Using any of these formats, a story including 
the associated photos and audio clips can be saved in a file. The file may then 
be uploaded to a web server and assigned a unique URL. The URL may then be 
20 sent on the web to specified recipient or list of recipients by email using the 
standard email protocols, for example SMTP. In one embodiment, to send the 
electronic mail (email), the system provides a network interface, for example 
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Ethernet, modem, or wireless. Alternatively, the file may be sent to the 
recipient(s) as an email attachment using a standard attachment encoding 
technique, for example Multipurpose Internet Mail Extensions (MIME). 

In one embodiment, the email address of the recipients may be specified 
5 using the scroll left/right and the add (+)/remove (-) buttons. These buttons 
are used to navigate among letters and punctuation symbols displayed on the 
top track 105. An email address may be constructed on the bottom track 115. 
The middle track may be used to display previously saved email addresses. 
Email addresses of people who send stories to the users may automatically be 
fi 10 added to the second track. Furthermore, the system may accept input email 

sT:: 

if* addresses through a "contact" file or vCard file (a Personal Data Interchange 

III (PDI) technology developed by the Versit Consortium) beamed by IR from a 

iL PDA (e.g., a Palm Pilot). To generate a new email address, the user may use a 

Lp M ll p hi! 

Pi stylus to type on an on-screen keyboard. Using the stylus, handwriting 

□ 15 recognition software, for example Graffiti by Palm Computing, may be used to 

..' .* T i fc * i u T7. 

recognize the user's handwriting. Alternatively, the user may use an external 
keyboard plugged into a keyboard interface provided by the system. Besides 
sending stories to email recipients, the system may also automatically check for 
new stories, for example, in the form of email messages or a predetermined web 
20 page listing new stories sent to the device/ owner, at set intervals or at set times 
of day. The system may also automatically download any new stories or 
messages. 
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In one embodiment, the system includes a cradle or docking station that 
includes power and network interface. In one embodiment, all communication 
(sending/receiving messages) is queued for execution when the system is 
docked. The communication then occurs automatically when docked. In one 
embodiment, while the system is in the cradle or otherwise unused for a set 
period of time, an auto play screen saver feature is activated causing the screen 
to cycle through all the stories in the system. 

In one embodiment, the system may create a movie file using the images 
and the associated audio clips in synchronization with each other, just as they 
are shown on the system when the user presses the play button. The movie file 
may then be stored in a standard format, such as, for example MPEG, AVI, or 
QuickTime, and may then be sent to the recipient(s) as an email attachment. In 
another embodiment, the author of a story may choose to send his or her own 
photograph to be associated with a story when the story is sent to a recipient. 
A link to this photograph is included and associated with the email. When the 
recipient views the email, the photograph of the author is displayed in an 
image slot. Alternatively, the photograph associated with the author might also 
be displayed as the first photograph in a story, or somehow combined with the 
first photograph especially for the collapsed view. 

As discussed above, the system may accept as input media objects from 
various sources. In one embodiment, other media objects, for example video 
clips, presentation slides, pages of documents, web pages, and audio clips may 
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also be loaded onto the system. With the video clips as input, an incoming 
video stream can be captured through a standard analog video capture board, 
or via a "firewire" (IEEE 1394) interface, or by direct connection to an already- 
digitized video file, for example a MPEG file, or a QuickTime file. The stream 
5 can then be segmented into clips using standard scene-change detection 

algorithms. Each clip can then be treated as a photograph. Furthermore, each 
clip may already have an associated audio segment from the original video 
source. In this embodiment, pressing the play button enables the system to 
play both the audio segment and the video clip synchronized with each other. 
10 The system then moves on to a next clip of the story. A first frame of the video 
clip may be used as a thumbnail image representing the video clip. 

With presentation slides as input, the system creates an image for each 
presentation slide. The presentation slides may have been generated by 
presentation software (e.g., Powerpoint) or other document or graphics editing 
15 tools. The story is then viewed on the system as a sequence of slides, similar to 
a presentation. With the pages of document as input, a story in the top track 
corresponds to the pages of an original document, in order. A story in the 
middle track is a collection of reorganized pages with added audio 
commentary, for example a summary. Documents can easily be imported from 
20 existing formats, both page-based, for example Postscript, or PDF, and linear, 
for example HTML, or plain text. In the latter case a pagination operation 
would be required. 
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With Web pages as input, the top track 115 may contain a user's 
chronological browsing history. This may be captured by monitoring a Web 
browser or by acting as a proxy server intercepting and serving all of a user's 
HTTP requests. Each story corresponds to a length of time (e.g., one day or one 
5 browsing session, etc.). The second track allows the user to compose and 

sequence groups of Web pages for easy access and later retrieval. In the second 
track, a story might correspond to a bookmark folder, a series of pages found in 
the course of one search, or any other structure as is useful to the user. 

With the audio clips as input, the top track may represent "albums" or 
10 audio CDs that the user has scanned into the system via a CD-ROM drive, or 
downloaded from the Web directly as a series of audio files, for example MP3 
files from MP3.com, Windows Media Audio files from Microsoft, Liquid Audio 
files from LiquidAudio.com, Realjukebox files from Real Networks, etc. Each 



album" is an ordered set of "songs". The second track represents the user's 



15 "playlists" (e.g., the user's own sequences of songs to be played). In this case, 
there is a default visual representation for each song, or the user chooses 
relevant icons, or a Web service provides icons to match artist names, song 
titles or album titles. 

Figure 7 illustrates an embodiment of an exemplary computer system 

20 that comprises the storage, display and interaction components of the digital 
story creation and play back system. The various components shown in Figure 
7 are provided by way of example. Certain components of the computer in 
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Figure 7 can be deleted from the system for a particular implementation of the 
invention. In other systems, additional components may be added without 
affecting the scope of the present invention. In other systems, additional 
components may be added without affecting the scope of the present invention. 

Figure 7 illustrates a system bus 700 to which various components are 
coupled. A processor 701 performs processing tasks required by the computer. 
Processor 701 may be any type of processing device capable of implementing 
the steps necessary to perform the storage, displaying, and interaction 
capabilities described herein. An input/output (I/O) device 704 is coupled to 
bus 700 for communicating with other devices coupled to the computer. A 
read-only memory (ROM) 703 and a random access memory (RAM) 705 are 
coupled to bus 700 to provide storage for various data and information used by 
the computer. Although ROM 703 and RAM 705 are shown coupled to bus 700, 
in alternate embodiments, ROM 703 and RAM 705 are coupled directly to 
processor 701 or coupled to a dedicated memory bus (not shown). 

A video display 710 is coupled to bus 700 and displays various 
information and data to the user of the computer. A disk drive 702 is coupled 
to bus 700 to provide long-term mass storage of information. In one 
embodiment, a pointing device 708 is coupled to bus 700 for entering 
information and commands to the computer system. In another embodiment 
two displays may be used, one low resolution display for the 3 tracks and a 
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high-resolution display for the main viewing area. The main viewing area may 
not be needed for audio and other media. 

In another embodiment, the multimedia story telling system is 
comprised of a console and one or more media cards. For example, the console 
may be a Nintendo console by Nintendo of America, Inc, and the media card 
may be a flash memory card capable of storing a large number of digital 
images. Figure 8 illustrates an exemplary console that can be used as a digital 
story telling system. One or more controllers 808 may be attached to the 
console 805 to accept input from the user. In one embodiment, the console 805 
is capable of providing all the control functions provided by the digital story 
telling system (shown in Figure 2). For example, the control functions on the 
digital story telling system are mapped to the console 805 such that the scroll 
left function on the system may be performed by moving the joystick on the 
controller 808 to the left, and the play function on the digital story telling 
system may be performed by pressing a <C> button on the controller 808, etc. 

Figure 9 is an exemplary table illustrating one embodiment of the 
mapping of the functions of the control buttons in the digital photo album to 
the joystick and buttons in the game console. In one embodiment, one or more 
game controllers is attached to the game console. The controller includes a 
joystick and multiple buttons (e.g., A, B, C, D, START, etc.). Using the table 900, 
the joystick can be used to navigate in the left, right, up, or down direction to 
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reach a media object on, for example, the third track. By pressing the <A> 
button on the controller, the media object is added to the first track. 

Referring back to Figure 8, in another embodiment, a display 810 is used 
to display the digital media normally displayed on the screen of the digital 
story telling system. The display 810 may be a monitor, a television, etc., and 
can allow multiple users to view the images in the stories at the same time. The 
display 810 may be connected to the console 806 through a display interface, 
such as, for example, a coaxial connection. Alternatively, the connection 
between the console 805 and the display 810 may be wireless. 

In one embodiment, the console 805 has a slot 806 to accept cartridges 
812. Each cartridge 812 has a slot 814 for inserting the media card to provide 
additional digital images to the console 805. For example, digital camera 
systems may use the media card to store captured digital images. The digital 
images may then be transferred from the media card to the console 805 through 
the cartridge 812. In another embodiment, the cartridge 812 itself may have a 
long-term storage capability such that when it is inserted into the console 805, 
the digital images in the media card are transferred to the long-term storage 
device in the cartridge 812. Alternatively, the images from the digital cameras 
may be transferred directly to the cartridge 812 through a direct interface on the 
cartridge 812 or through a connection on the console 805. The images stored on 
the cartridge 812 can then be displayed in the system using the three track 
display. 
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In one embodiment, the images associated with the stories created using 

the console 805 may not have audio annotations since the console may not have 

a recording capability. In this way, using the console and the control functions 

mapped from the system to the console, the images displayed on the tracks can 

5 be organized into stories without the annotation aspects of the stories. 

Alternatively, audio annotations can be added by editing these stories and 

using the recording capability of the digital story telling system at a later time. 

For example, the stories created by using the console 805 can be saved in the 

media card. The media card can then be used with the digital story telling 

J3 10 system and the audio annotations can be added to the stories. In another 

'i embodiment, the console 805 may allow the user to select one of multiple pre- 

m recorded background music selections to accompany the stories created from 

% the console 805. 

In one embodiment, some of the features of each image such as, for 

*m I* *i 

S 1 5 example, the background music is stored along with the image in the storage 

*H' m hl|llH 

device. The storage device may be the media card. The media card may have 
different storage capabilities providing the user different alternatives to store 
the images depending how many images the different media cards may hold. 
In another embodiment, the console 805 is Internet ready. With the 
20 appropriate communication interface such as, for example, a modem and the 
communication software such as, for example, an Internet browser, the console 
can be used to send stories to a recipient using the Internet. For example, the 
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user may use the console to enter the email address of the recipient. In one 
embodiment, when the user selects the option to send the story, the console 
enters into a send mode. In this mode, the three track arrangement may be 
used to specify the email address. For example, the top track may contain 
5 letters of the alphabet and the user select the letters similar to the way the user's 
name is entered before playing a game. The middle track may contain known 
email addresses, and the bottom track may contain a list of recipients. The 
joystick and the buttons can be used to select the address similar to the way the 
media object is selected. In one embodiment, to send the image to the recipients 

it""""; 

O 10 shown on the bottom track, the user may select the send option again. 

** H « 

'^Z? **! 

^ From the above description and drawings, it will be understood by those 

ry of ordinary skill in the art that the particular embodiments shown and 

s described are for purposes of illustration only and are not intended to limit the 

fj scope of the invention. Those of ordinary skill in the art will recognize that the 

SI 15 invention may be embodied in other specific forms without departing from its 
spirit or essential characteristics. References to details of particular 
embodiments are not intended to limit the scope of the claims. 
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CLAIMS 

What is claimed is: 

1. A system comprising: 

means for accepting a cartridge having a slot for a memory device to store 
a plurality of media objects; 

means for displaying the plurality of media objects in reduced visual 
representations in at least one track; and 

means for navigating among the reduced visual representations. 

2. The system of claim 1 wherein the plurality of media objects stored in the 
memory device are transferred to the cartridge when the memory device is 
inserted into the slot. 

3. The system of claim 2 wherein the memory device is a flash memory card. 

4. The system of claim 1 wherein the display means displays imported objects 
in a first track, authored stories in a second track and in a third track, at 
least one of the authored stories on the third track being part of a story 
under creation. 

5. The system of claim 4, wherein the imported objects are provided from the 
memory device and comprise the imported stories. 
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6. The system of claim 4, wherein means for navigating among the reduced 
visual representations comprises: 

means for selecting the media object from the first track and/ or from the 
second track; 

means for adding a representation of the selected media object to the third 
track to enable a user to author a new story; and 

means for moving one of the authored stories from the second track to the 
third track to enable the user to edit the authored story. 

7. The system of claim 6 further comprising means for selecting background 
music from a plurality of pre-recorded background music songs and means 
for associating the selected background music with the authored story. 

8. The system of claim 7, wherein the means for selecting comprises a game 
controller having a joystick and a plurality of control buttons. 

9. The system of claim 8, wherein the joystick is used to scroll among the first 
track, the second track and the third track, and wherein the plurality of 
control buttons comprise an add button and a delete button for adding and 
deleting, respectively, the media object in the third track. 
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10. 



The system of claim 9, wherein the joystick and the plurality of control 
buttons specify an Internet email address to send the plurality of media 
objects to a recipient. 



1 1 . The system of claim 1 0, wherein the plurality of control buttons and 
joystick specify the Internet email address by selecting a plurality of 
alphanumeric characters with the plurality of control buttons. 



12. The system of claim 10, wherein the plurality of control buttons and 

joystick select the Internet email address from a list of previously generated 
Internet email addresses. 



13. The system of claim 12, wherein the second track displays previously 
generated Internet email addresses, and wherein the third track displays 
the selected Internet email address. 



14. The system of claim 12, wherein the memory device is a flash memory 
card. 



15. A method comprising: 

accepting a cartridge having a slot for a media card storing a plurality of 
media objects; 

displaying the plurality of media objects in reduced visual representations 
displayed in a plurality of tracks; and 



navigating among the reduced visual representations. 
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16. The method of claim 15, further comprising transferring the plurality of 
media objects stored in the media card to the cartridge when the media 
card is inserted into the slot. 

17. The method of claim 16, wherein the media card comprises a flash memory 
card. 

18. The method of claim 15, wherein displaying the plurality of media objects 
comprises displaying imported stories in a first track on a display, 
authored stories in a second track and in a third track on the display, an 
authored story on the third track being a story under creation. 

19. The method of claim 18, further comprising transferring the imported 
stories from the media card, wherein the imported stories comprise of one 
or more media objects. 

20. The method of claim 15, wherein navigating among the reduced visual 
representations comprises: 

moving the media object from the first track and/or from the second track 
to the third track to enable a user to author a new story; and 

moving the authored story from the second track to the third track to 
enable the user to edit the authored story. 
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21 . The method of claim 20 further comprising selecting background music 
from a plurality of pre-recorded background music to associate the selected 
background music with the authored story. 

22. The method of claim 20, wherein moving the media object comprises using 
a game controller having a joystick and a plurality of control buttons. 

23. The method of claim 22, further comprising using the joystick and the 
plurality of control buttons to specify an Internet email address of a 
recipient to send the plurality of media objects. 

24. The method of claim 23, wherein specifying the Internet email address 
comprises selecting a plurality of alphanumeric characters. 

25. The method of claim 23, wherein specifying the Internet email address 
comprises selecting the Internet email address from a list of previously 
generated Internet email addresses. 

26. The method of claim 23, further comprising the second track displaying 
previously generated Internet email addresses, and the third track 
displaying the selected Internet email address. 

27. The method of claim 15, further comprising displaying the plurality of 
media objects on a television screen. 
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28. A system comprising 

a storage device to store a plurality of stories, each story comprising a 
plurality of media objects; and 

a console comprising 

a processor in communication with the storage device; and 

an input interface for accepting the storage device, the storage device 
having a slot for a media card comprising the plurality of media 
objects. 

29. The system of claim 28 further comprising a display coupled with the 
console, the display displaying the media objects in a first track, a second 
track and a third track. 

30. The system of claim 28 further comprising a controller, coupled with the 
console having a joystick and multiple control buttons to move media 
objects among the first track, the second track and the third track. 

31. The system of claim 30 wherein the controller controls play of a plurality of 
authored stories, each authored story comprising the plurality of media 
objects, wherein the authored story generated by the user using the 
controller to group related media objects. 
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32. The system of claim 31, wherein the display displays the authored stories 
in the second track and in the third track, with the authored story in the 
third track is under construction. 

33. The system of claim 28, wherein the media card comprises a flash memory 
card. 

34. A system, comprising 

a console having an input interface for accepting a storage device to store a 
plurality of media objects, the storage device having a slot for a media 
card; 

a display coupled with the console to display the plurality of media objects 
in a first track, a second track and a third track; and 

at least one controller coupled with the console, the controller comprising a 
joystick and at least one control button to navigate the plurality of 
media objects among the first track, the second track and the third 
track. 

35. The system of claim 34, wherein the media card comprises a flash memory 
storing the plurality of media objects. 

36. The system of claim 34, wherein the controller controls play of a plurality 
of authored stories, each authored story comprising the plurality of media 
objects, wherein the authored story generated by the user using the 
controller to group related media objects. 

Ricoh 4 1 07445 LP119X 



1 37. The system of claim 36, wherein the display displays the plurality of 

2 authored stories in the second track and in the third track, with the 

3 authored story in the third track is under construction. 
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ABSTRACT OF THE DISCLOSURE 

A method and a system that combine capabilities for storing, authoring, 
and viewing various forms of digital media are described. In one embodiment, 
the system comprises means for accepting a cartridge having a slot for a 
memory device to store a plurality of media objects, means for displaying the 
plurality of media objects in reduced visual representations in at least one track, 
and means for navigating among the reduced visual representations. 
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DIGITAL PHOTO ALBUM GAME CONSOLE 
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PATENT 



DECLARATION AND POWER OF ATTORNEY FOR PATENT APPLICATION 
(CONTINUATION-IN-PART) 

As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below, next to my name. 

I believe I am the original, first, and sole inventor (if only one name is listed below) or an original, first, 
and joint inventor (if plural names are listed below) of the subject matter which is claimed and for 
which a patent is sought on the invention entitled GAME CONSOLE BASED DIGITAL PHOTO ALUM 



the specification of which 

x is attached hereto. 

was filed on as 

United States Application Number 

or PCT International Application Number 

and was amended on . 

(if applicable) 

I hereby state that I have reviewed and understand the contents of the above-identified specification, 
including the claims, as amended by any amendment referred to above. 

I acknowledge the duty to disclose all information known to me to be material to patentability as 
defined in Title 37, Code of Federal Regulations, Section 1.56. 

I hereby claim foreign priority benefits under Title 35, United States Code, Section 1 19(a)-(d), of any 
foreign application(s) for patent or inventor's certificate listed below and have also identified below any 
foreign application for patent or inventor's certificate having a filing date before that of the application 
on which priority is claimed: 

Priority 

Prior Foreign Application(s) Claimed 



(Number) 


(Country) 


(Day/Month/Year Filed) 


Yes 


No 


(Number) 


(Country) 


(Day/Month/Year Filed) 


Yes 


No 


(Number) 


(Country) 


(Day/Month/Year Filed) 


Yes 


No 


I hereby claim the benefit under title 35, United States Code, Section 1 19(e) of any United States 
provisional application(s) listed below: 


Application Number 


Filing Date 








Application Number 


Filing Date 
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I hereby claim the benefit under Title 35. United States Code, Section 120 of any United States 
a pplication(s) listed below and, insofar as the subject matter of each of the claims of this 
application is not disclosed in the prior United States application in the manner provided by 
the first paragraph of Title 35, United States Code, Section 112, 1 acknowledge the duty to 
disclose all information known to me to be material to patentability as defined in Title 37, Code 
of Federal Regulations, Section 1.56 which became available between the filing date of the 
prior application and the national or PCT international filing date of this application: 

09/465,982 December 16, 1999 Pending 

Application Number Filing Date Status - patented, 

pending, abandoned 



Application Number Filing Date Status patented, 

pending, abandoned 



I hereby appoint the persons listed on Appendix A hereto (which is incorporated by reference and a 
part of this document) as my respective patent attorneys and patent agents, with full power of 
substitution and revocation, to prosecute this application and to transact all business in the Patent 
and Trademark Office connected herewith. 

Send correspondence to Michael J. Mallie , BLAKELY, SOKOLOFF, TAYLOR & 

(Name of Attorney or Agent) 
ZAFMAN LLP, 12400 Wilshire Boulevard 7th Floor, Los Angeles, California 90025 and direct 

telephone calls to Michael J. Mallie , (408) 720-8300. 

(Name of Attorney or Agent) 
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I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 



Full Name of Sole/First Inventor Gregory J. Wolff 



Inventor's Signature Date 

Residence Citizenship 

(City, State) (Country) 

Post Office Address 



Full Name of Second/Joint Inventor Marko Balabanovic 



Inventor's Signature Date 

Residence Citizenship 

(City, State) (Country) 

Post Office Address 



Full Name of Third/Joint Inventor Makoto Tanaka 



Inventor's Signature Date ________ 

Residence Citizenship 

(City, State) (Country) 

Post Office Address 



Full Name of Fourth/Joint Inventor 



Inventor's Signature Date 

Residence Citizenship 

(City, State) (Country) 

Post Office Address 
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APPENDIX A 



William E. Alford, Reg. No. 37,764; Farzad E. Amini, Reg. No. P42,261; Aloysius T. C. AuYeung, Reg. No. 
35,432; William Thomas Babbitt, Reg. No. 39,591 ; Carol F. Barry, Reg. No. 41,600; Jordan Michael 
Becker, Reg. No. 39,602; Bradley J. Bereznak, Reg. No. 33,474; Michael A. Bernadicou, Reg. No. 35,934; 
Roger W. Blakely, Jr., Reg. No. 25,831; Gregory D. Caldwell, Reg. No. 39,926; Ronald C. Card, Reg. No. 
44,587; Andrew C. Chen, Reg. No. 43,544; Thomas M. Coester, Reg. No. 39,637; Alin Corie, Reg. No. 
P46,244; Dennis M. deGuzman, Reg. No. 41,702; Stephen M. De Klerk, under 37 C.F.R. § 10.9(b); 
Michael Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew Diehl, 
Reg. No. 40,992; Sanjeet Dutta, Reg. No. P46,145; Matthew C. Fagan, Reg. No. 37,542; Tarek N. Fahmi, 
Reg. No. 41,402; Paramita Ghosh, Reg. No. 42,806; James Y. Go, Reg. No. 40,621; James A. Henry, 
Reg. No. 41,064; Willmore F. Holbrow III, Reg. No. P41,845; Sheryl Sue Holloway, Reg. No. 37,850; 
George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 30,139; William W. Kidd, Reg. No. 
31 ,772; Sang Hui Kim, Reg. No. 40,450; Eric T. King, Reg. No. 44,188; Erica W. Kuo, Reg. No. 42,775; 
Kurt P. Leyendecker, Reg. No. 42,799; Michael J. Mallie, Reg. No. 36,591; Andre L. Marais, under 37 
C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 42,879; Darren J. Milliken, Reg. 42,004; Lisa A. Norris, 
Reg. No. 44,976; Chun M. Ng, Reg. No. 36,878; Thien T. Nguyen, Reg. No. 43,835; Thinh V. Nguyen, 
Reg. No. 42,034; Dennis A. Nicholls, Reg. No. 42,036; Daniel E. Ovanezian, Reg. No. 41,236; Marina 
Portnova, Reg. No. P45,750; Babak Redjaian, Reg. No. 42,096; William F. Ryann, Reg. 44,313; James 
H. Salter, Reg. No. 35,668; William W. Schaal, Reg. No. 39,018; James C. Scheller, Reg. No. 31,195; 
Jeffrey Sam Smith, Reg. No. 39,377; Maria McCormack Sobrino, Reg. No. 31 ,639; Stanley W. Sokoloff, 
Reg. No. 25,128; Judith A. Szepesi, Reg. No. 39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. 
Taylor, Reg. No. 25,129; John F. Travis, Reg. No. 43,203; George G. C. Tseng, Reg. No. 41 ,355; Joseph 
A. Twarowski, Reg. No. 42,191; Lester J. Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 
41 ,364; John Patrick Ward, Reg. No. 40,216; Mark L. Watson, Reg. No. P46,322; Thomas C. Webster, 
Reg. No. P46,154; Charles T. J. Weigell, Reg. No. 43,398; Kirk D. Williams, Reg. No. 42,229; James M. 
Wu, Reg. No. 45,241; Steven D. Yates, Reg. No. 42,242; and Norman Zafman, Reg. No. 26,250; my 
patent attorneys, and Justin M. Dillon, Reg. No. 42,486; my patent agent, of BLAKELY, SOKOLOFF, 
TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th Floor, Los Angeles, 
California 90025, telephone (310) 207-3800, and James R. Thein, Reg. No. 31,710, my patent attorney. 
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APPENDIX B 



Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be material to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be material to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1 .98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1 ) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 



0) 
(ii) 



Opposing an argument of unpatentability relied on by the Office, or 
Asserting an argument of patentability. 



A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 



(D 
(2) 



Each inventor named in the application; 

Each attorney or agent who prepares or prosecutes the application; and 



(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 
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